Throughout the 1990s and into this century, the need for high-performance communications for server-to-storage and server-to-server networking has garnered much attention in the communications industry. Performance improvements in storage, processors and workstations, along with the move to distributed architectures such as client/server, have spawned increasingly data-intensive and high-speed networking applications.
Fiber Channel is a serial data transfer architecture standardized according to the American National Standards Institute (ANSI) Standard ANSI X3.230. Fiber Channel accommodates the fast transfer of large volumes of data between desktop workstations, mass storage subsystems, peripherals and host systems within a campus sized area. Fiber Channel offers a standard interface capable of simultaneously supporting both network and channel connections using multiple data communication protocols and is able to provide a number of benefits over traditional small computer system interface (SCSI) communications. For example, Fiber Channel permits faster speed, up to 10 to 250 times faster than typical local area network (LAN) speeds. In addition, Fiber Channel permits the connection of more devices and allows connections between devices over large distances. Fiber Channel has made the largest impact in the storage arena, using SCSI as an upper layer protocol. Further, Fiber Channel supports multiple data rates, media types and connectors while combining channel attributes with those of a LAN to provide a single interface capable of supporting both channel and network connections.
Channel and network protocols typically rely on buffers to hold transmitted and received data. The Fiber Channel protocol accommodates the transfer of data between the sending buffer at the source device (e.g., a computer or disk array) and the receiving buffer at the destination device. Moreover, the Fiber Channel protocol is not dependent on individual protocols implemented prior to receipt into or transmission from the buffers.
Flow control is an important aspect of Fiber Channel communications. Flow control refers to the management of data exchanges between two devices in a network to avoid data transmission loss upon congestion. If a device receives data faster than the device can process the data, data may be dropped or ignored. Similarly, data is lost if the storage capacity of the receiving device is not sufficient to accommodate the received data.
Utilizing flow control, a sending device transmits data to a receiving device only when the receiving device is ready to accept the data. Prior to sending the data, the communicating devices are initialized with respect to each other. This initialization includes the establishment of buffer-to-buffer credits (BBCs). The number of BBCs represents the number of data packets (e.g., frames) a sending device can transmit at one time without receiving an acknowledgement from the receiving device that the receiving device is available to receive at least one additional data packet. The BBC value is provided by the receiving device to the sending device during initialization. If each device has the capability to send data and receive data, a single BBC value is established for both devices. For example, the first device can indicate that it will accept up to four frames from the second device and the second device can indicate that it will accept up to eight frames from A. If determined independently, the BBC value of the first device would be eight and the BBC value of the second device would be four. However, the Fiber Channel protocol requires that both devices be assigned the lower BBC value of four. After enough data are transmitted from a device so that its BBC value is reached, no additional data are transmitted from that device. Data transmission from the sending device resumes when the receiving device indicates that it is ready to receive additional data, such as when the receiving device has processed at least a portion of the received data or buffer storage is again available. This procedure prevents receiver buffer overflow, thus avoiding loss of data.
Each device monitors the use of the available credits. The number of used credits (BBC_Credit_CNT) is initialized at zero prior to the initial transfer of frames between devices. BBC_Credit_CNT is incremented by one each time a frame is sent. Thus the number of remaining credits, representing the number of frames that can be sent to the other device without acknowledgement, is reduced by one. If BBC_Credit_CNT is greater than zero, BBC_Credit_CNT is decremented by 1 for each receiver ready primitive signal (R_RDY signal) received from the receiving device. Transmission of an R_RDY signal indicates the receiving device has processed a frame, made available a receive buffer and is ready to receive an additional frame. If BBC_Credit_CNT increases to equal BBC, the sending device is not allowed to transmit another frame until it receives an R_RDY signal.
Fiber Channel networks are inherently limited by latency. Latency is the transmission time for a data packet or signal to propagate between two nodes in a network. The latency generally increases as the distance between the nodes increases. In a SONET network, each kilometer of channel distance adds approximately 5 microseconds of transmission latency corresponding to the time for an optical pulse to travel through an optical channel between the two nodes.
FIG. 1 is a block diagram illustrating a Fiber Channel communication channel 10 in which a sending device 14 is transmitting data frames 16 to a receiving device 18. Each frame 16 received at the receiving device is buffered, if necessary, and an R_RDY signal is transmitted back to the sending device 14 when the frame 16 is transferred out of the buffer. Because the distance between the sending device 14 and the receiving device 18 is small in this example, the latency is insignificant and the data transmission rate (throughput) of the channel 10 is unaffected.
If the distance between the devices 14, 18 is sufficiently large, the latency reduces the average data transmission rate of the channel 10. FIGS. 2A to 2C illustrate the transmission of frames of data over a channel at three different times. The sending device 14 and the receiving device 18 are separated by a distance d2 substantially greater than the separation illustrated in FIG. 1. In FIG. 2A, data frames 16 (only four shown for clarity) are transmitted from the sending device 14 until its BBC_Credit_CNT equals its BBC value. At a later time as illustrated in FIG. 2B, the sending device 14 waits to receive an R_RDY signal from the receiving device 18 before sending its next frame 16. The receiving device 18 transmits an R_RDY signal back to the sending device 14 as each frame 16 is cleared from the buffer of the receiving device 18. Because of the length d2 of the separation, the sending device 14 continues to wait for the R_RDY signal even though the receiving device 18 has transmitted R_RDY signals and is available to receive additional frames. In FIG. 2C, the sending device 14 begins to receive the group of R_RDY signals and, therefore, resumes transmission of data frames 16. As a result of the propagation delay of the data frames and R_RDY signals, the data throughput is interrupted. The duration of the interruptions increases as the separation d2 of the devices 14, 18 increases.
The interruption, or “downtime”, resulting from latency is unacceptable in many types of networks including, for example, Storage Area Networks (SANs). If the distance between the two devices 14, 18 is sufficiently large as illustrated in FIGS. 2A to 2C, the maximum throughput possible without latency cannot be realized. The achievable throughput T is defined as   T  =            BBC      *      FS        RT  in which FS is the frame size in bits and RT is the round trip transmission time (i.e., round trip delay). A frame size of 2148 bytes i.e., a maximum transmission unit (MTU) for a Fiber Channel link, takes approximately 20 microseconds to transmit, which represents a distance of 4 km, or 2 km round-trip. Thus each BBC allows approximately 2 km of channel distance d2 without degrading the data throughput.
FIGS. 3 and 4 depict the relationship between throughput rate and channel length for a range of BBC values. The frame size is 2148 bytes. The BBC values are indicated in the legend in the upper right portion of the figures. The horizontal axis represents the length of the channel in kilometers and the vertical axis represents the throughput of the channel in Mb/s. If the BBC value is small (FIG. 3), the achievable throughput for the channel is substantially less than the maximum possible throughput for channel lengths exceeding a few kilometers. For example, for a BBC value of four, the achievable throughput is less than maximum possible throughput if the channel length is greater than 8 km. Larger BBC values (FIG. 4) allow greater channel lengths without a reduction in the throughput. For example, for a BBC value of 64, the achievable throughput is less than maximum possible throughput for channel lengths exceeding approximately 130 km. Thus the impact of latency on the data throughput is reduced by increasing the BBC values for the devices. This capacity increase, however, results in a significant increase in the cost of each device.
It is therefore desirable to provide a system and method for flow control in a Fiber Channel network that avoids unwanted delay time in data frame transmission without creating channel congestion, thereby extending the reach (i.e., the distance over which high data transfer rates can be maintained) of the data communication channel.